Typed processes #6368

bentsherman · 2025-08-27T17:56:51Z

This PR introduces a new syntax for process which uses typed inputs and outputs. The existing syntax is still supported.

This PR refactors several large classes -- namely ProcessConfig and TaskProcessor -- to better separate concerns and enable a v1 / v2 model for process inputs/outputs. When moving existing code to new files, I try to change it as little as possible to not break anything.

ProcessConfig refactor

The following new classes were spun out of ProcessConfig:

ProcessConfigV1 / ProcessConfigV2 extend ProcessConfig with the declared inputs / outputs based on legacy (v1) or typed (v2) semantics
ProcessDslV1 / ProcessDslV2 are builder DSLs for legacy / typed process definitions
ProcessConfigBuilder is an adapter for applying process configuration to a process definition
ProcessBuilder is the base builder class used by the above builders

TaskProcessor refactor

The following new classes were spun out of TaskProcessor:

TaskInputResolver implements the input file resolution from makeTaskContextStage2()
TaskOutputResolver implements the task output resolution logic for typed processes
TaskEnvCollector implements the output env/eval resolution from collectOutEnvMap()
TaskFileCollector implements the output file resolution from collectOutFiles()

Typed inputs / outputs

The following new classes implement the new behavior for typed inputs / outputs:

ProcessInputs and ProcessOutputs replace InputsList and OutputsList from the v1 model
ProcessInput and ProcessOutput replace all InParam and OutParam classes from the v1 model
ProcessFileInput and ProcessFileOutput replace FileInParam and FileOutParam in the v1 model

Backwards compatibility

The runtime supports both legacy (v1) and typed (v2) processes by creating the ProcessDef with either a ProcessConfigV1 or ProcessConfigV2.

ProcessDef, TaskProcessor, and TaskRun check this type to determine whether to use v1 or v2 semantics. An instanceof check is performed at these decision points:

if( config instanceof ProcessConfigV1 )
    // use legacy inputs/outputs
if( config instanceof ProcessConfigV2 )
    // use typed inputs/outputs

Based on initial work in #4553

TODO:

update docs
update tests
add e2e tests

netlify · 2025-08-27T17:56:58Z

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	`742de80`
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/68f11f0ed962ad0008befafe

christopher-hakkaart

Looking really good. I like the tutorial in particular. I think it's clear and includes the right amount of detail. Also, the order makes sense and it's a good length.

I will take a second pass and nit pick the language. In the meantime, I've added two high level comments. They are very minor.

docs/migrations/25-10.md

docs/tutorials/static-types.md

pditommaso

This a too big change compared to current syntax. I do not support this approach

docs/migrations/25-10.md

docs/tutorials/static-types.md

bentsherman · 2025-09-05T20:45:57Z

Updated to use "phase 1" syntax, i.e. support for multiple input channels and tuple inputs

christopher-hakkaart

@bentsherman - I went through the tutorial in detail focusing on the language. I split everything into separate comments to hopefully make it easier to accept/reject.

I found some of the code blocks confusing as when I went into the example repo the code blocks didn't match what was in master branch. I'm fine with using the rnaseq-nf example and a little bit of difference is okay, but if anyone does what I tried to do it's hard to follow. Can we better align this? Alternatively, can we peel off this example from rnaseq-nf and start building an repo full of examples specifically for the docs? If might give a little more latitude for v1, v2, v3 of tutorials like this and allow better synergy between what is written and what's in the repo. Happy to hear your thoughts

docs/tutorials/static-types.md

christopher-hakkaart

Looking really good!

I've made suggestions to thin words in some places.

I've also made suggestions to change some of the language in definition lists (what I messaged about earlier). This might stick out too much while the rest of the page is still what a developer does vs what the method does, i.e., "Return a" vs "Returns".

See what you think in the context of the page. If it sticks out too much reject and I'll make a PR later to standardize the page.

docs/migrations/25-10.md

docs/process-typed.md

docs/migrations/25-10.md

docs/reference/process.md

docs/reference/stdlib-types.md

docs/reference/syntax.md

bentsherman · 2025-10-10T14:11:23Z

Thanks Chris. I'm happy to go ahead with the recommend language for the typed process reference and we can update the rest of that page later on.

christopher-hakkaart

Thanks Ben, readings the docs again (particularly the guides and process-typed page), I have a few more suggestions:

Standardize language around the Nextflow VS Code extension. Maybe it can be a state it's Nextflow VS Code extension once, and then referred to as VS Code extension afterwards. I think it could be hammered as the Nextflow VS Code extension for clarity, but not a sticking point either way.
Framing in the migration guide. I think it could be more explanation, then steps for migration. I think a repeated structure and language here helps explain then instruct, make it clearer overall about what the changes are and how to implement them.
Reduce admonitions at the top of the page. It's a little jarring looking at the preview and seeing 3 admonitions before one sentence and a code block. I tried to make this text to make it a smoother transition into the page.

None of these are sticking points, so I've approved from a docs perspective. Once these are merged. There is some language on the migration notes page I would like to update, but I won't make a PR until after this is merged to keep down conflicts (although there probably won't be many).

docs/tutorials/static-types.md

docs/process-typed.md

bentsherman · 2025-10-13T16:23:37Z

@claude please review

claude · 2025-10-13T16:23:54Z

Claude finished @bentsherman's task —— View job

Comprehensive Code Review

This is an excellent PR that introduces typed processes to Nextflow with a well-thought-out architecture. Here's my detailed analysis:

Architecture & Design ✅

ProcessConfig Refactoring: The refactoring of ProcessConfig into V1/V2 variants is exemplary:

Clean separation of concerns with minimal code duplication
ProcessConfigV1 handles legacy InputsList/OutputsList
ProcessConfigV2 handles new typed ProcessInputs/ProcessOutputs
Base class remains focused on core configuration logic

TaskProcessor Changes: Good extraction of responsibilities:

TaskInputResolver - handles file input resolution logic
TaskOutputResolver - handles typed output resolution
TaskEnvCollector - environment variable collection
TaskFileCollector - output file collection with proper input exclusion

Typed Input/Output Model: Well-designed type system:

ProcessInputs/ProcessOutputs provide clean interfaces
Support for nullable types with Path? syntax
Proper separation of parameters, environment, files, and stdin/topics
Good use of composition over inheritance

Backwards Compatibility ✅

The backwards compatibility approach is solid:

Runtime checks with instanceof ProcessConfigV1/V2 at decision points (TaskProcessor.groovy:1510-1514)
Legacy processes continue to work unchanged
V1/V2 models coexist cleanly without interference

Documentation & Tutorials ✅

Excellent documentation with:

Comprehensive tutorial in docs/tutorials/static-types.md with practical rnaseq-nf example
Complete reference documentation in docs/process-typed.md
Clear migration guide with specific examples
Good coverage of edge cases (nullable types, staging directives)

Test Coverage ✅

Strong test coverage including:

New unit tests for all extracted classes (TaskFileCollectorTest, TaskInputsResolverTest, etc.)
Integration tests with typed examples (collect-tuple-typed.nf, env-typed.nf, etc.)
Tests cover core functionality and edge cases

Implementation Quality ✅

Code Quality:

Clean separation of V1/V2 logic paths
Proper error handling and validation
Good use of Groovy idioms and existing patterns
Comprehensive type checking in new classes

Parser/AST Changes:

Grammar extensions in ScriptParser.g4 look correct for new syntax
AST nodes properly separated into V1/V2 variants
Visitor pattern implementation maintains consistency

Potential Areas for Improvement

Error Messages: Consider enhancing error messages for type mismatches to be more user-friendly, especially for newcomers to typed processes.
Performance: While the instanceof checks are minimal overhead, consider if there are hot paths where this could be optimized with a strategy pattern in the future.
Documentation: Consider adding more examples of complex nested types and how they interact with existing Nextflow operators.

Minor Issues

Line docs/tutorials/static-types.md:45 has a typo: "nextflow_scham.json" should be "nextflow_schema.json"
Consider adding validation for unsupported type combinations early in the parsing phase

Overall Assessment

This is a well-executed feature addition that:

Maintains full backwards compatibility
Introduces powerful new capabilities
Has excellent test coverage and documentation
Uses sound software engineering principles
Follows established Nextflow patterns and conventions

The PR successfully delivers on its goals of enabling static type checking while preserving the existing user experience. The architecture is extensible and should serve as a solid foundation for future enhancements.

Recommendation: Approve - This is high-quality work that significantly enhances Nextflow's capabilities while maintaining its core principles.

bentsherman · 2025-10-13T16:59:11Z

Regarding Claude's comments:

Error Messages: Consider enhancing error messages for type mismatches to be more user-friendly, especially for newcomers to typed processes.

Type checking errors will be implemented in the language server for now.

Performance: While the instanceof checks are minimal overhead, consider if there are hot paths where this could be optimized with a strategy pattern in the future.

Not opposed in principle, but likely overkill for now. In the final version we could end up with something like TaskProcessorV1/V2 instead, but we'll see.

Documentation: Consider adding more examples of complex nested types and how they interact with existing Nextflow operators.

We will continue to expand the docs, just wanted to add the basics with this PR.

bentsherman · 2025-10-13T16:59:39Z

@pditommaso let me know if you have any remaining concerns or things you want to test before we merge

pditommaso · 2025-10-14T15:55:21Z

This is huge 😬 Please, make a build using [e2e prod]

bentsherman · 2025-10-14T19:34:39Z

e2e tests passed 👍

modules/nextflow/src/main/groovy/nextflow/processor/TaskFileCollector.groovy

pditommaso

Massive work, some little things to address

modules/nextflow/src/main/groovy/nextflow/processor/TaskEnvCollector.groovy

modules/nextflow/src/main/groovy/nextflow/processor/TaskOutputResolver.groovy

modules/nextflow/src/main/groovy/nextflow/script/BaseScript.groovy

modules/nextflow/src/main/groovy/nextflow/script/params/v2/ProcessFileInput.groovy

modules/nextflow/src/main/groovy/nextflow/script/params/v2/ProcessFileOutput.groovy

modules/nextflow/src/main/groovy/nextflow/script/params/v2/ProcessInputsDef.groovy

modules/nextflow/src/main/groovy/nextflow/script/params/v2/ProcessOutputsDef.groovy

modules/nextflow/src/main/groovy/nextflow/script/params/v2/ProcessInputs.groovy

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

Feedback was addressed

bentsherman added the lang/processes label Aug 27, 2025

bentsherman mentioned this pull request Aug 27, 2025

Static types for process inputs/outputs #4553

Closed

13 tasks

bentsherman changed the title ~~Typed processe~~ Typed processes Aug 27, 2025

bentsherman marked this pull request as ready for review September 1, 2025 17:53

bentsherman requested review from a team as code owners September 1, 2025 17:53

bentsherman added this to the 25.10 milestone Sep 1, 2025

bentsherman force-pushed the typed-processes branch from f7dd01a to 71526ad Compare September 2, 2025 20:41

christopher-hakkaart reviewed Sep 2, 2025

View reviewed changes

docs/migrations/25-10.md Outdated Show resolved Hide resolved

docs/tutorials/static-types.md Outdated Show resolved Hide resolved

pditommaso requested changes Sep 3, 2025

View reviewed changes

mahesh-panchal reviewed Sep 3, 2025

View reviewed changes

docs/migrations/25-10.md Outdated Show resolved Hide resolved

docs/tutorials/static-types.md Outdated Show resolved Hide resolved

bentsherman force-pushed the typed-processes branch 3 times, most recently from 25a80b1 to 0e7be56 Compare September 5, 2025 20:44

christopher-hakkaart reviewed Sep 9, 2025

View reviewed changes

bentsherman mentioned this pull request Sep 9, 2025

Simplify process entry execution #6391

Closed

bentsherman requested a review from pditommaso September 10, 2025 15:13

bentsherman force-pushed the typed-processes branch from af09783 to 814d40c Compare October 9, 2025 02:55

bentsherman requested a review from christopher-hakkaart October 9, 2025 02:59

bentsherman linked an issue Oct 9, 2025 that may be closed by this pull request

DSL2 - emit tuples with optional values #2678

Closed

bentsherman mentioned this pull request Oct 9, 2025

Publish fails if it encounters an empty Value channel #6458

Open

pditommaso force-pushed the master branch from b7b4221 to c1114bc Compare October 9, 2025 20:41

christopher-hakkaart reviewed Oct 10, 2025

View reviewed changes

christopher-hakkaart approved these changes Oct 12, 2025

View reviewed changes

bentsherman mentioned this pull request Oct 13, 2025

Type checking support #6482

Draft

bentsherman force-pushed the typed-processes branch from 822340a to 7a46926 Compare October 14, 2025 16:21

pditommaso reviewed Oct 15, 2025

View reviewed changes

modules/nextflow/src/main/groovy/nextflow/processor/TaskFileCollector.groovy Outdated Show resolved Hide resolved

pditommaso previously requested changes Oct 15, 2025

View reviewed changes

bentsherman requested a review from pditommaso October 15, 2025 22:30

Typed processes [e2e prod]

742de80

Signed-off-by: Ben Sherman <bentshermann@gmail.com>

bentsherman force-pushed the typed-processes branch from 0ace872 to 742de80 Compare October 16, 2025 16:36

bentsherman merged commit 30f3c3c into master Oct 17, 2025
24 checks passed

bentsherman deleted the typed-processes branch October 17, 2025 15:27

Typed processes #6368

Typed processes #6368

Uh oh!

Conversation

bentsherman commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ProcessConfig refactor

TaskProcessor refactor

Typed inputs / outputs

Backwards compatibility

Uh oh!

netlify bot commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for nextflow-docs-staging canceled.

Uh oh!

christopher-hakkaart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

pditommaso left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

bentsherman commented Sep 5, 2025

Uh oh!

christopher-hakkaart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

christopher-hakkaart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bentsherman commented Oct 10, 2025

Uh oh!

christopher-hakkaart left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

bentsherman commented Oct 13, 2025

Uh oh!

claude bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Comprehensive Code Review

Architecture & Design ✅

Backwards Compatibility ✅

Documentation & Tutorials ✅

Test Coverage ✅

Implementation Quality ✅

Potential Areas for Improvement

Minor Issues

bentsherman commented Aug 27, 2025 •

edited

Loading

netlify bot commented Aug 27, 2025 •

edited

Loading

claude bot commented Oct 13, 2025 •

edited

Loading